Automated Detection of Martian Dune Fields Using a Convolutional Neural Network

A workflow by Cole Speed and Yiran Shen, University of Texas at Austin

Aeolian sand dunes are prevalent landforms on the surface of Mars. While attempts have been made to map their distribution in satellite images, it is likely that many remain unmapped. In this project, we seek to develop a method to automatically identify dune fields on the surface of mars using deep learning. Specifically, we hope to achieve this through an image segmentation approach use of a convolutional neural (UNet) via a collection of images captured by the Mars Reconnaissance Orbiter.

Left. Image of the Red Planet (Credit: NASA). Right. False-color image shows 'sea' of dunes near Mars' northern polar cap (Credit: NASA).

Left. Image captured by the Mars Reconnaisance Orbiter, shows barchan dunes (Credit: NASA/JPL/University of Arizona/USGS).Right. First dunes studied up-close, known as "High Dune" on NASA's Curiosity rover (Credit:NASA/JPL-Caltech).

Background and Significance of Dune Fields on Mars

Formation of Dunes on Mars and Variety of Dunes on the Martian Surface

Deposits on the surface of Mars provide the only available record of its temporal evolution. Many relict features remain well preserved on the Martian surface, documenting periods of actively flowing water and Martian wind regimes. One such landform-type are aeolian dunes, which form and evolve due to wind-driven sediment transport. A variety of dune types including those commonly seen on Earth (barchan, linear, cresentic) are also observed on the Martian surface. However, a variety of dune geometries exist on Mars that are not found on Earth. The distribution, configuration, and types of dunes on Mars provide history of changes on the Red Planet. NASA has even developed a way of monitoring dune movement throught time across the Martian surface!

Goal: Dune Field Detection and Classification

We would like to develop an automated approach for detecting and classifying martian dune fields through the application of deep learning and image segmentation. We will do so through the use of a convolutional neural network trained using hand-mapped dune field locations and high-resolution images captured by the THEMIS IR camera on the Mars Reconnaisance Orbitor.

Available Datasets

Imagery

The Mars Reconnaisance Orbitor provides a rich dataset of visible imagery and thermal data from the Martian surface. The Thermal Emission Imaging System (THEMIS) IR camera affixed to the Mars Reconnaisance Orbiter provides the data used in this study. A global mosaic of THEMIS imagery sampled at 100x100m per pixel resolution is the primary dataset for model training, validation, and testing.

Left. Mars Reconnaisance Orbiter, THEMIS camera imagery acquisition (Credit: NASA). Right. Global THEMIS Mosaic has 100x100m per pixel spatial resolution (Credit: NASA).

Hand-mapped Dune Polygons

Previous workers have mapped the locations of dune fields as polygons based on visual inspection of imagery from the Mars Reconnaisance Oribiter. These data and information on the location of mapped Martian dune fields have been compiled into the Mars Global Digital Dune Database: MC2–MC29. However, many dune fields on the Martian surface remain unmapped.

Previous work: Deep Learning Applied to Planetary Surface Processes

Deep learning, and specifically convolutional neural networks, in the scope of planetary surfaces and surface processes has been applied primarily to automatically identify and measure craters on both the Moon and Mars (Lee, 2019; Silbert et al., 2019). However, automatic detection and classification of dune fields using CNNs has not been achieved.

We decided to use the UNet architecture to approach our problem after seeing its success in identifying and mapping craters on the Moon and Mars (DeepMoon and DeepMars Github repositories)(Lee, 2019; Silbert et al., 2019), which proved successful to determine the positions and sizes of craters from Lunar and Martian digital elevation maps.

Their UNet convolutional neural network (and by design our neural network) is provided in the schematic below.

In our project we implement a custom version of the UNET architecture (Ronneberger et al., 2015). This architecture consists of a contracting path (left side) and expansive path (right side), joined through multi-level skip connections (middle). Martian THEMIS images are input to the contracting path and predictions are output from a final layer following the expansive path.

In the above schematic, boxes represent cross-sections of square feature maps. Each map's dimensions are indicated on its lower left, and its number of channels are indicated above it. In this diagram, the leftmost map is a 256 × 256 grayscale image sampled from the digital elevation map of the Moon, and the rightmost the CNN's binary ring mask prediction. We employ the same approach for highlighting regions that correspond to dune fields in the THEMIS imagery.

Exploring the Data and Data Preprocessing

Prior to model construction, we visualize the data using mapping software (QGIS). We see that the dunes are strikingly different than the landscape surrounding them. We also notice that many dune fields are located within craters.

Types of dunes seen in the THEMIS Imagery

Dune fields are clearly imaged by the THEMIS IR camera. Because the images are high-resolution (100x100m per pixel) individual dunes are clearly distinguishable.

However, we observe a wide degree of variability in dune field geometry and prevalence. We will have to see if the CNN can identify this variability!

Training data: Satellite imagery from the Mars Reconnaisance Orbiter

Training images were derived from the THEMIS Day IR 100m Global Mosaic, which provides imagery across the entire Martian surface at 100 meters per pixel. The full imagery dataset is ~22Gb

Label data

Hand-mapped dune fields constitute the label data for this study. These data were obtained from the Global Mars Dune Database, in which 550 dune fields have been mapped by hand based on visual interpretation of satellite imagery.

Data preprocessing

A significant amount of time for this project went into the data collection, cleaning and preparation.

We first reprojected the label shapefiles in QGIS to ensure consistent coordinate reference systems between the labels and images. Mars Simple Cylindrical coordinates were used.

A binary raster was generated from the label feature class where dune field polygon pixels have values of 1 while other pixels have Null values.

Due of the file size of the original THEMIS mosaic (~20Gb), we first cropped the satellite image and label rasters to a region which included the majority of the mapped dune field locations See Above.

We then generated image and label 'tiles' with dimensions of 256x256 pixels using a series of python scrips in combination with the open-source GNU IMAGE MANIPULATION PROGRAM (GIMP). These scripts can be found on Github.

This workflow resulted in 78,337 image and label tiles. For training purposes, we filtered the image tiles to include only images that contained at least one (or a portion of one) dune field. This resulted in a total dataset of 1116 image/label pairs.

Importing and visualizing the data

The input data file mars_dunes.npz (about 1Gb) is available at https://drive.google.com/file/d/1mBn8mQ1O51EprTpXSNQtK-VSmF3wD7Z-/view?usp=sharing

Using Google Drive, you can create a shortcut for a shared file:

1. Right click the file.
2. Click Add shortcut to Drive.

from google.colab import drive
drive.mount('/content/gdrive')

Mounted at /content/gdrive

import numpy as np

loaded = np.load('/content/gdrive/My Drive/mars_dunes.npz')
data = loaded['data']
labels = loaded['labels']

print(data.shape,labels.shape)

(1116, 256, 256) (1116, 256, 256)

The are 1116 two-dimensional training images with dimensions $256 \times 256$. We have only included image tiles that have labels, which excludes a large portion of images tiles. Below, we visualize a random sampling of image tiles to better understand our training dataset.

import matplotlib.pyplot as plt 
import random

def plot_mars_image(data,title):
    random.seed(2000)
    
    fig, axs = plt.subplots(nrows=5, ncols=5, figsize=(20, 12),
                            subplot_kw={'xticks': [], 'yticks': []})
    for ax in axs.flat:
        pick = random.randint(0,data.shape[0]-1)
        ax.imshow(data[pick,:,:],cmap='gray')
    plt.tight_layout()
    fig.suptitle(title,fontsize=20)
    plt.subplots_adjust(top=0.9)
    plt.show()
    
plot_mars_image(data,'Training Data: THEMIS Images Tiles (256 x 256 pixels)')

We can also look at the corresponding labels that will be used in the training and validation of our neural network. We visualize the labels corresponding to the same random sampling of dune fields below.

def plot_dune(data,title):
    fig, axs = plt.subplots(nrows=5, ncols=5, figsize=(20, 12),
                            subplot_kw={'xticks': [], 'yticks': []})
    random.seed(2000)
    for ax in axs.flat:
        pick = random.randint(0,data.shape[0]-1)
        ax.imshow(data[pick,:,:],cmap='gray_r',interpolation='bilinear')
    plt.tight_layout()
    fig.suptitle(title,fontsize=20)
    plt.subplots_adjust(top=0.9)
    plt.show()
    
plot_dune(labels,'Training Labels: Mars Global Dune Database Polygons')

Notice that most image/label tiles only capture a portion of a dune field (some only a very small portion). Below we visualize the image data and labels together.

def plot_mars_dune(data,labels,title):
    fig, axs = plt.subplots(nrows=5, ncols=5, figsize=(20, 12),
                            subplot_kw={'xticks': [], 'yticks': []})
    random.seed(2000)
    for ax in axs.flat:
        pick = random.randint(0,data.shape[0]-1)
        ax.imshow(data[pick,:,:],cmap='gray')
        # overlay fault labels with alpha for transparency
        ax.imshow(labels[pick,:,:],cmap='gnuplot',alpha=0.5)
    plt.tight_layout()
    fig.suptitle(title,fontsize=20)
    plt.subplots_adjust(top=0.9)
    plt.show()
    
plot_mars_dune(data,labels,'Training Data: THEMIS Images and Dune Field Polygons Plotted Together')

Preparing data for training

To prepare the data for training, we will first normalize each image by subtracting the mean value and dividing by the standard deviation, resulting in each image having a mean of 0 and a standard deviation of 1. A fourth dimension of 1 is added to indicate that the input dataset has one channel.

# normalize data
mean = data.mean(axis=(1,2))[:,np.newaxis,np.newaxis]
std = data.std(axis=(1,2))[:,np.newaxis,np.newaxis]
data_normal = (data-mean)/std

#add dimension of 1
data_normal = np.expand_dims(data_normal,axis=3)
data_normal.shape

(1116, 256, 256, 1)

We randomly split the data for training (80%) and validation (20%).

from sklearn.model_selection import train_test_split

# Split the data for training and validation
data_train,data_valid,labels_train,labels_valid = \
train_test_split(data_normal,labels,test_size=0.2,shuffle= True)

Designing the convolutional neural network (Unet)

We are going to use a convolutional neural network (CNN) accelerated on GPUs.

When used Google Colab for this project, utilizing their GPU for accelerating the training of our CNN.

from tensorflow import config

# setup GPU card
gpus = config.list_physical_devices('GPU')

if gpus:
    for gpu in gpus:
        print(gpu)
        # dynamic memory allocation
        config.experimental.set_memory_growth(gpu,True)

PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')

Now we can design our Unet-based neural network.

from tensorflow import keras
from tensorflow.keras import layers

nb_filter = [16,32,64,128,256]
def Unet(nb_filter):
    keras.backend.clear_session()
    
    # can take arbitrary input size
    image = keras.Input((None,None,1),name='input')
    
    conv0 = layers.Conv2D(nb_filter[0], (3,3), padding='same')(image)
    conv0 = layers.BatchNormalization()(conv0)
    conv0 = layers.Activation('relu')(conv0)
    conv0 = layers.Conv2D(nb_filter[0], (3,3), padding='same')(conv0)
    conv0 = layers.BatchNormalization()(conv0)
    conv0 = layers.Activation('relu')(conv0)

    pool0 = layers.MaxPooling2D(pool_size=(2,2))(conv0)

    conv1 = layers.Conv2D(nb_filter[1], (3,3), padding='same')(pool0)
    conv1 = layers.BatchNormalization()(conv1)
    conv1 = layers.Activation('relu')(conv1)
    conv1 = layers.Conv2D(nb_filter[1], (3,3), padding='same')(conv1)
    conv1 = layers.BatchNormalization()(conv1)
    conv1 = layers.Activation('relu')(conv1)
    
    pool1 = layers.MaxPooling2D(pool_size=(2,2))(conv1)

    conv2 = layers.Conv2D(nb_filter[2], (3,3), padding='same')(pool1)
    conv2 = layers.BatchNormalization()(conv2)
    conv2 = layers.Activation('relu')(conv2)
    conv2 = layers.Conv2D(nb_filter[2], (3,3), padding='same')(conv2)
    conv2 = layers.BatchNormalization()(conv2)
    conv2 = layers.Activation('relu')(conv2)
    
    pool2 = layers.MaxPooling2D(pool_size=(2,2))(conv2)

    conv3 = layers.Conv2D(nb_filter[3], (3,3), padding='same')(pool2)
    conv3 = layers.BatchNormalization()(conv3)
    conv3 = layers.Activation('relu')(conv3)
    conv3 = layers.Conv2D(nb_filter[3], (3,3), padding='same')(conv3)
    conv3 = layers.BatchNormalization()(conv3)
    conv3 = layers.Activation('relu')(conv3)
    
    pool3 = layers.MaxPooling2D(pool_size=(2,2))(conv3)

    conv4 = layers.Conv2D(nb_filter[4], (3,3), padding='same')(pool3)
    conv4 = layers.BatchNormalization()(conv4)
    conv4 = layers.Activation('relu')(conv4)
    conv4 = layers.Conv2D(nb_filter[4], (3,3), padding='same')(conv4)
    conv4 = layers.BatchNormalization()(conv4)
    conv4 = layers.Activation('relu')(conv4)

    up5 = layers.concatenate([layers.UpSampling2D(size=(2,2))(conv4), conv3], axis=-1)

    conv5 = layers.Conv2D(nb_filter[3], (3,3), padding='same')(up5)
    conv5 = layers.BatchNormalization()(conv5)
    conv5 = layers.Activation('relu')(conv5)
    conv5 = layers.Conv2D(nb_filter[3], (3,3), padding='same')(conv5)
    conv5 = layers.BatchNormalization()(conv5)
    conv5 = layers.Activation('relu')(conv5)

    up6 = layers.concatenate([layers.UpSampling2D(size=(2,2))(conv5), conv2], axis=-1)

    conv6 = layers.Conv2D(nb_filter[2], (3,3), padding='same')(up6)
    conv6 = layers.BatchNormalization()(conv6)
    conv6 = layers.Activation('relu')(conv6)
    conv6 = layers.Conv2D(nb_filter[2], (3,3), padding='same')(conv6)
    conv6 = layers.BatchNormalization()(conv6)
    conv6 = layers.Activation('relu')(conv6)

    up7 = layers.concatenate([layers.UpSampling2D(size=(2,2))(conv6), conv1], axis=-1)

    conv7 = layers.Conv2D(nb_filter[1], (3,3), padding='same')(up7)
    conv7 = layers.BatchNormalization()(conv7)
    conv7 = layers.Activation('relu')(conv7)
    conv7 = layers.Conv2D(nb_filter[1], (3,3), padding='same')(conv7)
    conv7 = layers.BatchNormalization()(conv7)
    conv7 = layers.Activation('relu')(conv7)

    up8 = layers.concatenate([layers.UpSampling2D(size=(2,2))(conv7), conv0], axis=-1)
    
    conv8 = layers.Conv2D(nb_filter[0], (3,3), padding='same')(up8)
    conv8 = layers.BatchNormalization()(conv8)
    conv8 = layers.Activation('relu')(conv8)
    conv8 = layers.Conv2D(nb_filter[0], (3,3), padding='same')(conv8)
    conv8 = layers.BatchNormalization()(conv8)
    conv8 = layers.Activation('relu')(conv8)

    conv9 = layers.Conv2D(1, (1,1), activation='sigmoid')(conv8)
    
    model = keras.Model(inputs=[image], outputs=[conv9])

    return model

We take a look at the model constuction, below.

model=Unet(nb_filter)
model.summary()

Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input (InputLayer)              [(None, None, None,  0                                            
__________________________________________________________________________________________________
conv2d (Conv2D)                 (None, None, None, 1 160         input[0][0]                      
__________________________________________________________________________________________________
batch_normalization (BatchNorma (None, None, None, 1 64          conv2d[0][0]                     
__________________________________________________________________________________________________
activation (Activation)         (None, None, None, 1 0           batch_normalization[0][0]        
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, None, None, 1 2320        activation[0][0]                 
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, None, None, 1 64          conv2d_1[0][0]                   
__________________________________________________________________________________________________
activation_1 (Activation)       (None, None, None, 1 0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
max_pooling2d (MaxPooling2D)    (None, None, None, 1 0           activation_1[0][0]               
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, None, None, 3 4640        max_pooling2d[0][0]              
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, None, None, 3 128         conv2d_2[0][0]                   
__________________________________________________________________________________________________
activation_2 (Activation)       (None, None, None, 3 0           batch_normalization_2[0][0]      
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, None, None, 3 9248        activation_2[0][0]               
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, None, None, 3 128         conv2d_3[0][0]                   
__________________________________________________________________________________________________
activation_3 (Activation)       (None, None, None, 3 0           batch_normalization_3[0][0]      
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)  (None, None, None, 3 0           activation_3[0][0]               
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, None, None, 6 18496       max_pooling2d_1[0][0]            
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, None, None, 6 256         conv2d_4[0][0]                   
__________________________________________________________________________________________________
activation_4 (Activation)       (None, None, None, 6 0           batch_normalization_4[0][0]      
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, None, None, 6 36928       activation_4[0][0]               
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, None, None, 6 256         conv2d_5[0][0]                   
__________________________________________________________________________________________________
activation_5 (Activation)       (None, None, None, 6 0           batch_normalization_5[0][0]      
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)  (None, None, None, 6 0           activation_5[0][0]               
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, None, None, 1 73856       max_pooling2d_2[0][0]            
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, None, None, 1 512         conv2d_6[0][0]                   
__________________________________________________________________________________________________
activation_6 (Activation)       (None, None, None, 1 0           batch_normalization_6[0][0]      
__________________________________________________________________________________________________
conv2d_7 (Conv2D)               (None, None, None, 1 147584      activation_6[0][0]               
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, None, None, 1 512         conv2d_7[0][0]                   
__________________________________________________________________________________________________
activation_7 (Activation)       (None, None, None, 1 0           batch_normalization_7[0][0]      
__________________________________________________________________________________________________
max_pooling2d_3 (MaxPooling2D)  (None, None, None, 1 0           activation_7[0][0]               
__________________________________________________________________________________________________
conv2d_8 (Conv2D)               (None, None, None, 2 295168      max_pooling2d_3[0][0]            
__________________________________________________________________________________________________
batch_normalization_8 (BatchNor (None, None, None, 2 1024        conv2d_8[0][0]                   
__________________________________________________________________________________________________
activation_8 (Activation)       (None, None, None, 2 0           batch_normalization_8[0][0]      
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, None, None, 2 590080      activation_8[0][0]               
__________________________________________________________________________________________________
batch_normalization_9 (BatchNor (None, None, None, 2 1024        conv2d_9[0][0]                   
__________________________________________________________________________________________________
activation_9 (Activation)       (None, None, None, 2 0           batch_normalization_9[0][0]      
__________________________________________________________________________________________________
up_sampling2d (UpSampling2D)    (None, None, None, 2 0           activation_9[0][0]               
__________________________________________________________________________________________________
concatenate (Concatenate)       (None, None, None, 3 0           up_sampling2d[0][0]              
                                                                 activation_7[0][0]               
__________________________________________________________________________________________________
conv2d_10 (Conv2D)              (None, None, None, 1 442496      concatenate[0][0]                
__________________________________________________________________________________________________
batch_normalization_10 (BatchNo (None, None, None, 1 512         conv2d_10[0][0]                  
__________________________________________________________________________________________________
activation_10 (Activation)      (None, None, None, 1 0           batch_normalization_10[0][0]     
__________________________________________________________________________________________________
conv2d_11 (Conv2D)              (None, None, None, 1 147584      activation_10[0][0]              
__________________________________________________________________________________________________
batch_normalization_11 (BatchNo (None, None, None, 1 512         conv2d_11[0][0]                  
__________________________________________________________________________________________________
activation_11 (Activation)      (None, None, None, 1 0           batch_normalization_11[0][0]     
__________________________________________________________________________________________________
up_sampling2d_1 (UpSampling2D)  (None, None, None, 1 0           activation_11[0][0]              
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, None, None, 1 0           up_sampling2d_1[0][0]            
                                                                 activation_5[0][0]               
__________________________________________________________________________________________________
conv2d_12 (Conv2D)              (None, None, None, 6 110656      concatenate_1[0][0]              
__________________________________________________________________________________________________
batch_normalization_12 (BatchNo (None, None, None, 6 256         conv2d_12[0][0]                  
__________________________________________________________________________________________________
activation_12 (Activation)      (None, None, None, 6 0           batch_normalization_12[0][0]     
__________________________________________________________________________________________________
conv2d_13 (Conv2D)              (None, None, None, 6 36928       activation_12[0][0]              
__________________________________________________________________________________________________
batch_normalization_13 (BatchNo (None, None, None, 6 256         conv2d_13[0][0]                  
__________________________________________________________________________________________________
activation_13 (Activation)      (None, None, None, 6 0           batch_normalization_13[0][0]     
__________________________________________________________________________________________________
up_sampling2d_2 (UpSampling2D)  (None, None, None, 6 0           activation_13[0][0]              
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (None, None, None, 9 0           up_sampling2d_2[0][0]            
                                                                 activation_3[0][0]               
__________________________________________________________________________________________________
conv2d_14 (Conv2D)              (None, None, None, 3 27680       concatenate_2[0][0]              
__________________________________________________________________________________________________
batch_normalization_14 (BatchNo (None, None, None, 3 128         conv2d_14[0][0]                  
__________________________________________________________________________________________________
activation_14 (Activation)      (None, None, None, 3 0           batch_normalization_14[0][0]     
__________________________________________________________________________________________________
conv2d_15 (Conv2D)              (None, None, None, 3 9248        activation_14[0][0]              
__________________________________________________________________________________________________
batch_normalization_15 (BatchNo (None, None, None, 3 128         conv2d_15[0][0]                  
__________________________________________________________________________________________________
activation_15 (Activation)      (None, None, None, 3 0           batch_normalization_15[0][0]     
__________________________________________________________________________________________________
up_sampling2d_3 (UpSampling2D)  (None, None, None, 3 0           activation_15[0][0]              
__________________________________________________________________________________________________
concatenate_3 (Concatenate)     (None, None, None, 4 0           up_sampling2d_3[0][0]            
                                                                 activation_1[0][0]               
__________________________________________________________________________________________________
conv2d_16 (Conv2D)              (None, None, None, 1 6928        concatenate_3[0][0]              
__________________________________________________________________________________________________
batch_normalization_16 (BatchNo (None, None, None, 1 64          conv2d_16[0][0]                  
__________________________________________________________________________________________________
activation_16 (Activation)      (None, None, None, 1 0           batch_normalization_16[0][0]     
__________________________________________________________________________________________________
conv2d_17 (Conv2D)              (None, None, None, 1 2320        activation_16[0][0]              
__________________________________________________________________________________________________
batch_normalization_17 (BatchNo (None, None, None, 1 64          conv2d_17[0][0]                  
__________________________________________________________________________________________________
activation_17 (Activation)      (None, None, None, 1 0           batch_normalization_17[0][0]     
__________________________________________________________________________________________________
conv2d_18 (Conv2D)              (None, None, None, 1 17          activation_17[0][0]              
==================================================================================================
Total params: 1,968,225
Trainable params: 1,965,281
Non-trainable params: 2,944
__________________________________________________________________________________________________

Our model has 1,968,225 total parameters.

keras.utils.plot_model(model,show_shapes=True)

Model training

Now we are ready to train our model. As before, we will use orginal size (1116 image tiles) of the dataset, cross-entropy as the loss function (applied pixel-by-pixel in the image), and Adam as the optimization algorithm.

model0=Unet(nb_filter)
model0.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

# Fit the model
batch_size = 40
epochs = 50
fit0 = model0.fit(data_train,labels_train,
                validation_data=(data_valid,labels_valid),
                epochs=epochs,
                batch_size=batch_size)

Epoch 1/50
23/23 [==============================] - 53s 623ms/step - loss: 0.4721 - accuracy: 0.8088 - val_loss: 7.3836 - val_accuracy: 0.1154
Epoch 2/50
23/23 [==============================] - 10s 431ms/step - loss: 0.2846 - accuracy: 0.9144 - val_loss: 4.4437 - val_accuracy: 0.2130
Epoch 3/50
23/23 [==============================] - 10s 434ms/step - loss: 0.2620 - accuracy: 0.9112 - val_loss: 0.7318 - val_accuracy: 0.5912
Epoch 4/50
23/23 [==============================] - 10s 440ms/step - loss: 0.2530 - accuracy: 0.9107 - val_loss: 0.3456 - val_accuracy: 0.8982
Epoch 5/50
23/23 [==============================] - 10s 444ms/step - loss: 0.2320 - accuracy: 0.9180 - val_loss: 0.3209 - val_accuracy: 0.8949
Epoch 6/50
23/23 [==============================] - 10s 449ms/step - loss: 0.2108 - accuracy: 0.9268 - val_loss: 0.3087 - val_accuracy: 0.8939
Epoch 7/50
23/23 [==============================] - 10s 456ms/step - loss: 0.2090 - accuracy: 0.9261 - val_loss: 0.2653 - val_accuracy: 0.9021
Epoch 8/50
23/23 [==============================] - 11s 464ms/step - loss: 0.2219 - accuracy: 0.9183 - val_loss: 0.2815 - val_accuracy: 0.8977
Epoch 9/50
23/23 [==============================] - 11s 471ms/step - loss: 0.2169 - accuracy: 0.9202 - val_loss: 0.2975 - val_accuracy: 0.9016
Epoch 10/50
23/23 [==============================] - 11s 469ms/step - loss: 0.2042 - accuracy: 0.9265 - val_loss: 0.2469 - val_accuracy: 0.9181
Epoch 11/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1963 - accuracy: 0.9281 - val_loss: 0.2424 - val_accuracy: 0.9241
Epoch 12/50
23/23 [==============================] - 11s 458ms/step - loss: 0.1910 - accuracy: 0.9305 - val_loss: 0.2330 - val_accuracy: 0.9270
Epoch 13/50
23/23 [==============================] - 10s 457ms/step - loss: 0.1887 - accuracy: 0.9304 - val_loss: 0.2060 - val_accuracy: 0.9313
Epoch 14/50
23/23 [==============================] - 10s 457ms/step - loss: 0.1842 - accuracy: 0.9321 - val_loss: 0.2191 - val_accuracy: 0.9264
Epoch 15/50
23/23 [==============================] - 11s 459ms/step - loss: 0.1758 - accuracy: 0.9345 - val_loss: 0.2099 - val_accuracy: 0.9320
Epoch 16/50
23/23 [==============================] - 11s 462ms/step - loss: 0.1666 - accuracy: 0.9405 - val_loss: 0.2333 - val_accuracy: 0.9148
Epoch 17/50
23/23 [==============================] - 11s 464ms/step - loss: 0.1900 - accuracy: 0.9280 - val_loss: 0.1953 - val_accuracy: 0.9312
Epoch 18/50
23/23 [==============================] - 11s 464ms/step - loss: 0.1764 - accuracy: 0.9337 - val_loss: 0.1854 - val_accuracy: 0.9367
Epoch 19/50
23/23 [==============================] - 11s 464ms/step - loss: 0.1644 - accuracy: 0.9396 - val_loss: 0.1762 - val_accuracy: 0.9377
Epoch 20/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1605 - accuracy: 0.9398 - val_loss: 0.1978 - val_accuracy: 0.9284
Epoch 21/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1758 - accuracy: 0.9347 - val_loss: 0.2223 - val_accuracy: 0.8974
Epoch 22/50
23/23 [==============================] - 11s 462ms/step - loss: 0.1669 - accuracy: 0.9362 - val_loss: 0.2005 - val_accuracy: 0.9073
Epoch 23/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1533 - accuracy: 0.9424 - val_loss: 0.1647 - val_accuracy: 0.9368
Epoch 24/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1618 - accuracy: 0.9397 - val_loss: 0.1756 - val_accuracy: 0.9377
Epoch 25/50
23/23 [==============================] - 11s 462ms/step - loss: 0.1500 - accuracy: 0.9431 - val_loss: 0.1904 - val_accuracy: 0.9256
Epoch 26/50
23/23 [==============================] - 11s 459ms/step - loss: 0.1535 - accuracy: 0.9397 - val_loss: 0.2046 - val_accuracy: 0.9154
Epoch 27/50
23/23 [==============================] - 11s 460ms/step - loss: 0.1581 - accuracy: 0.9383 - val_loss: 0.2578 - val_accuracy: 0.9117
Epoch 28/50
23/23 [==============================] - 11s 462ms/step - loss: 0.1541 - accuracy: 0.9404 - val_loss: 0.1971 - val_accuracy: 0.9258
Epoch 29/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1375 - accuracy: 0.9487 - val_loss: 0.1740 - val_accuracy: 0.9342
Epoch 30/50
23/23 [==============================] - 11s 465ms/step - loss: 0.1278 - accuracy: 0.9523 - val_loss: 0.1828 - val_accuracy: 0.9293
Epoch 31/50
23/23 [==============================] - 11s 464ms/step - loss: 0.1312 - accuracy: 0.9506 - val_loss: 0.2019 - val_accuracy: 0.9131
Epoch 32/50
23/23 [==============================] - 11s 464ms/step - loss: 0.1218 - accuracy: 0.9532 - val_loss: 0.1753 - val_accuracy: 0.9377
Epoch 33/50
23/23 [==============================] - 11s 465ms/step - loss: 0.1266 - accuracy: 0.9514 - val_loss: 0.2507 - val_accuracy: 0.9304
Epoch 34/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1257 - accuracy: 0.9512 - val_loss: 0.1866 - val_accuracy: 0.9427
Epoch 35/50
23/23 [==============================] - 11s 462ms/step - loss: 0.1119 - accuracy: 0.9567 - val_loss: 0.1574 - val_accuracy: 0.9450
Epoch 36/50
23/23 [==============================] - 11s 462ms/step - loss: 0.1104 - accuracy: 0.9575 - val_loss: 0.2004 - val_accuracy: 0.9232
Epoch 37/50
23/23 [==============================] - 11s 461ms/step - loss: 0.1116 - accuracy: 0.9569 - val_loss: 0.1661 - val_accuracy: 0.9391
Epoch 38/50
23/23 [==============================] - 11s 463ms/step - loss: 0.1060 - accuracy: 0.9588 - val_loss: 0.1910 - val_accuracy: 0.9337
Epoch 39/50
23/23 [==============================] - 11s 461ms/step - loss: 0.0978 - accuracy: 0.9628 - val_loss: 0.1741 - val_accuracy: 0.9384
Epoch 40/50
23/23 [==============================] - 11s 461ms/step - loss: 0.0976 - accuracy: 0.9627 - val_loss: 0.1844 - val_accuracy: 0.9349
Epoch 41/50
23/23 [==============================] - 11s 462ms/step - loss: 0.1014 - accuracy: 0.9630 - val_loss: 0.2121 - val_accuracy: 0.9263
Epoch 42/50
23/23 [==============================] - 11s 461ms/step - loss: 0.0990 - accuracy: 0.9624 - val_loss: 0.2298 - val_accuracy: 0.9148
Epoch 43/50
23/23 [==============================] - 11s 463ms/step - loss: 0.0961 - accuracy: 0.9625 - val_loss: 0.1785 - val_accuracy: 0.9436
Epoch 44/50
23/23 [==============================] - 11s 463ms/step - loss: 0.0797 - accuracy: 0.9704 - val_loss: 0.1950 - val_accuracy: 0.9421
Epoch 45/50
23/23 [==============================] - 11s 463ms/step - loss: 0.0750 - accuracy: 0.9721 - val_loss: 0.1880 - val_accuracy: 0.9365
Epoch 46/50
23/23 [==============================] - 11s 464ms/step - loss: 0.0777 - accuracy: 0.9704 - val_loss: 0.2515 - val_accuracy: 0.9345
Epoch 47/50
23/23 [==============================] - 11s 462ms/step - loss: 0.0793 - accuracy: 0.9700 - val_loss: 0.2445 - val_accuracy: 0.9059
Epoch 48/50
23/23 [==============================] - 11s 463ms/step - loss: 0.0759 - accuracy: 0.9714 - val_loss: 0.1914 - val_accuracy: 0.9247
Epoch 49/50
23/23 [==============================] - 11s 463ms/step - loss: 0.0648 - accuracy: 0.9755 - val_loss: 0.1710 - val_accuracy: 0.9421
Epoch 50/50
23/23 [==============================] - 11s 464ms/step - loss: 0.0707 - accuracy: 0.9732 - val_loss: 0.2204 - val_accuracy: 0.9404

def plot_history(history):
    fig = plt.figure(figsize=(10,6))
    plt.plot(history['accuracy'],label='training')
    plt.plot(history['val_accuracy'],label='validation')
    plt.ylabel('Accuracy',fontsize=20)
    plt.xlabel('Epoch',fontsize=20)
    plt.legend(fontsize=20)
    plt.show()

    # summarize history for loss
    fig = plt.figure(figsize=(10,6))
    plt.plot(history['loss'],label='training')
    plt.plot(history['val_loss'],label='validation')
    plt.ylabel('Loss',fontsize=20)
    plt.xlabel('Epoch',fontsize=20)
    plt.legend(fontsize=20)
    plt.show()

plot_history(fit0.history)

Data augmentation: A larger training dataset and (hopefully) improved training

More data is always (usually?) better. We now extend the dataset to be four times larger than our orginal one by flipping and rotating the dataset.

# flipping data
f_data = np.flip(data,axis=1)
f_labels = np.flip(labels,axis=1)
# rotate data
r_data = np.flip(f_data,axis=0)
r_labels = np.flip(f_labels,axis=0)
# flipping again
h_data = np.flip(data,axis=0)
h_labels = np.flip(labels,axis=0)

t_data = np.concatenate([data, r_data, h_data, f_data], axis=0)
t_labels = np.concatenate([labels, r_labels, h_labels, f_labels], axis=0)
print(data.shape,labels.shape)
print(t_data.shape,t_labels.shape)

# normalize data
mean = t_data.mean(axis=(1,2))[:,np.newaxis,np.newaxis]
std = t_data.std(axis=(1,2))[:,np.newaxis,np.newaxis]
t_data_normal = (t_data-mean)/std

#add dimension of 1
t_data = np.expand_dims(t_data_normal,axis=3)
t_data.shape

(1116, 256, 256) (1116, 256, 256)
(4464, 256, 256) (4464, 256, 256)

(4464, 256, 256, 1)

# Split the data for training and validation
t_data_train,t_data_valid,t_labels_train,t_labels_valid = \
train_test_split(t_data,t_labels,test_size=0.2,shuffle= True)

t_data_train.shape

(3571, 256, 256, 1)

model=Unet(nb_filter)
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
# Fit the model
batch_size = 40
epochs = 50
fit = model.fit(t_data_train,t_labels_train,
                validation_data=(t_data_valid,t_labels_valid),
                epochs=epochs,
                batch_size=batch_size)

Epoch 1/50
90/90 [==============================] - 26s 264ms/step - loss: 0.5931 - accuracy: 0.7568 - val_loss: 0.6724 - val_accuracy: 0.5428
Epoch 2/50
90/90 [==============================] - 22s 245ms/step - loss: 0.3411 - accuracy: 0.9216 - val_loss: 0.3834 - val_accuracy: 0.8745
Epoch 3/50
90/90 [==============================] - 22s 245ms/step - loss: 0.2761 - accuracy: 0.9210 - val_loss: 0.2695 - val_accuracy: 0.9114
Epoch 4/50
90/90 [==============================] - 22s 245ms/step - loss: 0.2305 - accuracy: 0.9281 - val_loss: 0.2808 - val_accuracy: 0.9069
Epoch 5/50
90/90 [==============================] - 22s 245ms/step - loss: 0.2135 - accuracy: 0.9261 - val_loss: 0.2335 - val_accuracy: 0.9137
Epoch 6/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1951 - accuracy: 0.9320 - val_loss: 0.2940 - val_accuracy: 0.8656
Epoch 7/50
90/90 [==============================] - 22s 245ms/step - loss: 0.2034 - accuracy: 0.9256 - val_loss: 0.2175 - val_accuracy: 0.9216
Epoch 8/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1804 - accuracy: 0.9347 - val_loss: 0.1823 - val_accuracy: 0.9363
Epoch 9/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1702 - accuracy: 0.9380 - val_loss: 0.1855 - val_accuracy: 0.9324
Epoch 10/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1686 - accuracy: 0.9387 - val_loss: 0.1997 - val_accuracy: 0.9312
Epoch 11/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1665 - accuracy: 0.9381 - val_loss: 0.2422 - val_accuracy: 0.9018
Epoch 12/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1636 - accuracy: 0.9401 - val_loss: 0.1609 - val_accuracy: 0.9391
Epoch 13/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1442 - accuracy: 0.9473 - val_loss: 0.1566 - val_accuracy: 0.9423
Epoch 14/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1421 - accuracy: 0.9465 - val_loss: 0.1738 - val_accuracy: 0.9307
Epoch 15/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1465 - accuracy: 0.9446 - val_loss: 0.1778 - val_accuracy: 0.9372
Epoch 16/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1495 - accuracy: 0.9434 - val_loss: 0.1445 - val_accuracy: 0.9455
Epoch 17/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1349 - accuracy: 0.9500 - val_loss: 0.1468 - val_accuracy: 0.9453
Epoch 18/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1306 - accuracy: 0.9507 - val_loss: 0.1679 - val_accuracy: 0.9374
Epoch 19/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1235 - accuracy: 0.9528 - val_loss: 0.1404 - val_accuracy: 0.9447
Epoch 20/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1193 - accuracy: 0.9554 - val_loss: 0.1403 - val_accuracy: 0.9471
Epoch 21/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1087 - accuracy: 0.9588 - val_loss: 0.1601 - val_accuracy: 0.9369
Epoch 22/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1055 - accuracy: 0.9602 - val_loss: 0.1170 - val_accuracy: 0.9553
Epoch 23/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1067 - accuracy: 0.9596 - val_loss: 0.1468 - val_accuracy: 0.9419
Epoch 24/50
90/90 [==============================] - 22s 245ms/step - loss: 0.1062 - accuracy: 0.9601 - val_loss: 0.1505 - val_accuracy: 0.9400
Epoch 25/50
90/90 [==============================] - 22s 244ms/step - loss: 0.0972 - accuracy: 0.9632 - val_loss: 0.1921 - val_accuracy: 0.9332
Epoch 26/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0851 - accuracy: 0.9676 - val_loss: 0.1353 - val_accuracy: 0.9470
Epoch 27/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0793 - accuracy: 0.9701 - val_loss: 0.1012 - val_accuracy: 0.9622
Epoch 28/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0797 - accuracy: 0.9695 - val_loss: 0.1006 - val_accuracy: 0.9610
Epoch 29/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0688 - accuracy: 0.9735 - val_loss: 0.0966 - val_accuracy: 0.9640
Epoch 30/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0653 - accuracy: 0.9749 - val_loss: 0.1128 - val_accuracy: 0.9589
Epoch 31/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0610 - accuracy: 0.9764 - val_loss: 0.0983 - val_accuracy: 0.9649
Epoch 32/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0536 - accuracy: 0.9792 - val_loss: 0.0918 - val_accuracy: 0.9652
Epoch 33/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0702 - accuracy: 0.9726 - val_loss: 0.0827 - val_accuracy: 0.9698
Epoch 34/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0502 - accuracy: 0.9803 - val_loss: 0.0881 - val_accuracy: 0.9657
Epoch 35/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0442 - accuracy: 0.9826 - val_loss: 0.0913 - val_accuracy: 0.9693
Epoch 36/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0385 - accuracy: 0.9846 - val_loss: 0.5002 - val_accuracy: 0.9246
Epoch 37/50
90/90 [==============================] - 22s 244ms/step - loss: 0.0472 - accuracy: 0.9814 - val_loss: 0.0915 - val_accuracy: 0.9704
Epoch 38/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0377 - accuracy: 0.9849 - val_loss: 0.0700 - val_accuracy: 0.9756
Epoch 39/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0321 - accuracy: 0.9871 - val_loss: 0.0786 - val_accuracy: 0.9734
Epoch 40/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0337 - accuracy: 0.9865 - val_loss: 0.0767 - val_accuracy: 0.9744
Epoch 41/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0304 - accuracy: 0.9878 - val_loss: 0.0775 - val_accuracy: 0.9760
Epoch 42/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0288 - accuracy: 0.9883 - val_loss: 0.0680 - val_accuracy: 0.9778
Epoch 43/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0264 - accuracy: 0.9893 - val_loss: 0.0666 - val_accuracy: 0.9800
Epoch 44/50
90/90 [==============================] - 22s 244ms/step - loss: 0.0243 - accuracy: 0.9902 - val_loss: 0.0788 - val_accuracy: 0.9776
Epoch 45/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0272 - accuracy: 0.9890 - val_loss: 0.0701 - val_accuracy: 0.9796
Epoch 46/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0228 - accuracy: 0.9907 - val_loss: 0.0746 - val_accuracy: 0.9779
Epoch 47/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0367 - accuracy: 0.9854 - val_loss: 0.2604 - val_accuracy: 0.9288
Epoch 48/50
90/90 [==============================] - 22s 244ms/step - loss: 0.0604 - accuracy: 0.9763 - val_loss: 0.0811 - val_accuracy: 0.9746
Epoch 49/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0279 - accuracy: 0.9888 - val_loss: 0.0641 - val_accuracy: 0.9801
Epoch 50/50
90/90 [==============================] - 22s 244ms/step - loss: 0.0197 - accuracy: 0.9920 - val_loss: 0.0618 - val_accuracy: 0.9825

plot_history(fit.history)

Cross entropy balancing as the other loss function in our model

Since it is evident that the number of background shapes in any given detection image is usually more than the number of dune class instances, training a neural network might make it biased to outputting background class more compared to dune class and it may affects the performance of a neural net. The problem of class imbalance can be solved by adding more instances of the less dominant class in training data.

import tensorflow as tf
def _to_tensor(x, dtype):
    x = tf.convert_to_tensor(x)
    if x.dtype != dtype:
        x = tf.cast(x, dtype)
    return x

def cross_entropy_balanced(y_true, y_pred):
    from keras import backend as K
    _epsilon = _to_tensor(K.epsilon(), y_pred.dtype.base_dtype)
    y_pred   = tf.clip_by_value(y_pred, _epsilon, 1 - _epsilon)
    y_pred   = tf.math.log(y_pred/ (1 - y_pred))
    y_true = tf.cast(y_true, tf.float32)

    count_neg = tf.reduce_sum(1. - y_true)
    count_pos = tf.reduce_sum(y_true)
    beta = count_neg / (count_neg + count_pos)
    pos_weight = beta / (1 - beta)

    cost = tf.nn.weighted_cross_entropy_with_logits(logits=y_pred, labels=y_true, pos_weight=pos_weight)
    cost = tf.reduce_mean(cost * (1 - beta))

    return tf.where(tf.equal(count_pos, 0.0), 0.0, cost)

model1=Unet(nb_filter)
model1.compile(loss=cross_entropy_balanced,
              #optimizer='adam',
              optimizer=keras.optimizers.Adam(0.001),
              metrics=['accuracy'])
# Fit the model
batch_size = 40
epochs = 50
fit = model1.fit(t_data_train,t_labels_train,
                validation_data=(t_data_valid,t_labels_valid),
                epochs=epochs,
                batch_size=batch_size)

Epoch 1/50
90/90 [==============================] - 25s 253ms/step - loss: 0.0998 - accuracy: 0.7043 - val_loss: 0.1735 - val_accuracy: 0.7205
Epoch 2/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0763 - accuracy: 0.8120 - val_loss: 0.1947 - val_accuracy: 0.8844
Epoch 3/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0702 - accuracy: 0.8483 - val_loss: 0.1151 - val_accuracy: 0.9137
Epoch 4/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0697 - accuracy: 0.8491 - val_loss: 0.1371 - val_accuracy: 0.9171
Epoch 5/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0635 - accuracy: 0.8593 - val_loss: 0.0714 - val_accuracy: 0.8998
Epoch 6/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0613 - accuracy: 0.8663 - val_loss: 0.0761 - val_accuracy: 0.8762
Epoch 7/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0606 - accuracy: 0.8754 - val_loss: 0.0793 - val_accuracy: 0.9165
Epoch 8/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0614 - accuracy: 0.8732 - val_loss: 0.0662 - val_accuracy: 0.8716
Epoch 9/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0591 - accuracy: 0.8763 - val_loss: 0.0700 - val_accuracy: 0.8402
Epoch 10/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0563 - accuracy: 0.8840 - val_loss: 0.0614 - val_accuracy: 0.8721
Epoch 11/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0513 - accuracy: 0.8865 - val_loss: 0.1234 - val_accuracy: 0.9338
Epoch 12/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0537 - accuracy: 0.8871 - val_loss: 0.0608 - val_accuracy: 0.8621
Epoch 13/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0452 - accuracy: 0.9001 - val_loss: 0.0575 - val_accuracy: 0.8834
Epoch 14/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0470 - accuracy: 0.8939 - val_loss: 0.0500 - val_accuracy: 0.8961
Epoch 15/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0399 - accuracy: 0.9123 - val_loss: 0.0537 - val_accuracy: 0.8887
Epoch 16/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0421 - accuracy: 0.9065 - val_loss: 0.0533 - val_accuracy: 0.9206
Epoch 17/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0371 - accuracy: 0.9208 - val_loss: 0.0570 - val_accuracy: 0.8810
Epoch 18/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0348 - accuracy: 0.9251 - val_loss: 0.0609 - val_accuracy: 0.9353
Epoch 19/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0324 - accuracy: 0.9241 - val_loss: 0.0498 - val_accuracy: 0.9144
Epoch 20/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0382 - accuracy: 0.9076 - val_loss: 0.0528 - val_accuracy: 0.8750
Epoch 21/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0281 - accuracy: 0.9329 - val_loss: 0.0421 - val_accuracy: 0.9241
Epoch 22/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0281 - accuracy: 0.9326 - val_loss: 0.0518 - val_accuracy: 0.9464
Epoch 23/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0273 - accuracy: 0.9382 - val_loss: 0.0380 - val_accuracy: 0.9411
Epoch 24/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0218 - accuracy: 0.9501 - val_loss: 0.0421 - val_accuracy: 0.9395
Epoch 25/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0214 - accuracy: 0.9515 - val_loss: 0.0854 - val_accuracy: 0.9502
Epoch 26/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0224 - accuracy: 0.9512 - val_loss: 0.0402 - val_accuracy: 0.9225
Epoch 27/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0190 - accuracy: 0.9560 - val_loss: 0.0370 - val_accuracy: 0.9254
Epoch 28/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0179 - accuracy: 0.9569 - val_loss: 0.0453 - val_accuracy: 0.9517
Epoch 29/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0175 - accuracy: 0.9592 - val_loss: 0.0505 - val_accuracy: 0.9380
Epoch 30/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0173 - accuracy: 0.9583 - val_loss: 0.0332 - val_accuracy: 0.9512
Epoch 31/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0154 - accuracy: 0.9649 - val_loss: 0.0401 - val_accuracy: 0.9382
Epoch 32/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0189 - accuracy: 0.9567 - val_loss: 0.0353 - val_accuracy: 0.9576
Epoch 33/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0155 - accuracy: 0.9645 - val_loss: 0.0327 - val_accuracy: 0.9364
Epoch 34/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0139 - accuracy: 0.9657 - val_loss: 0.0320 - val_accuracy: 0.9616
Epoch 35/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0113 - accuracy: 0.9723 - val_loss: 0.0530 - val_accuracy: 0.9676
Epoch 36/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0115 - accuracy: 0.9717 - val_loss: 0.0304 - val_accuracy: 0.9632
Epoch 37/50
90/90 [==============================] - 22s 245ms/step - loss: 0.0110 - accuracy: 0.9734 - val_loss: 0.0314 - val_accuracy: 0.9475
Epoch 38/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0107 - accuracy: 0.9738 - val_loss: 0.0418 - val_accuracy: 0.9702
Epoch 39/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0123 - accuracy: 0.9716 - val_loss: 0.1152 - val_accuracy: 0.9506
Epoch 40/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0186 - accuracy: 0.9591 - val_loss: 0.0351 - val_accuracy: 0.9585
Epoch 41/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0108 - accuracy: 0.9726 - val_loss: 0.0338 - val_accuracy: 0.9697
Epoch 42/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0087 - accuracy: 0.9792 - val_loss: 0.0348 - val_accuracy: 0.9696
Epoch 43/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0075 - accuracy: 0.9819 - val_loss: 0.0328 - val_accuracy: 0.9671
Epoch 44/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0084 - accuracy: 0.9792 - val_loss: 0.0342 - val_accuracy: 0.9681
Epoch 45/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0068 - accuracy: 0.9838 - val_loss: 0.0400 - val_accuracy: 0.9749
Epoch 46/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0063 - accuracy: 0.9848 - val_loss: 0.0372 - val_accuracy: 0.9719
Epoch 47/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0091 - accuracy: 0.9786 - val_loss: 0.0329 - val_accuracy: 0.9651
Epoch 48/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0107 - accuracy: 0.9743 - val_loss: 0.0292 - val_accuracy: 0.9672
Epoch 49/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0086 - accuracy: 0.9798 - val_loss: 0.0379 - val_accuracy: 0.9766
Epoch 50/50
90/90 [==============================] - 22s 246ms/step - loss: 0.0062 - accuracy: 0.9849 - val_loss: 0.0348 - val_accuracy: 0.9731

plot_history(fit.history)

Save the model

We save our model termed "deepdunes_model.h5", so that it can be reload throught the Google driver for further testing.

#Save model
keras.models.save_model(model,'deepdunes_model.h5',save_format='h5')
# #Load the model back with model
# model = keras.models.load_model('deepdunes_model.h5')

Checking the results compared to the hand-mapped dune field locations

prediction = model.predict(t_data_valid)

# thresholding probability
labels_pred = prediction
labels_pred[prediction>=0.5]=1
labels_pred[prediction<0.5]=0

plot_dune(t_data_valid[:,:,:,0],'Example Validation Data')

plot_dune(t_labels_valid,'Example Validation Labels')

plot_dune(labels_pred[:,:,:,0],'Predicted Dune Field Labels')

plot_mars_dune(t_data_valid[:,:,:,0],labels_pred[:,:,:,0],'Predicted Labels')

Testing on full image dataset to identify additional dune field candidates

We would like to know if there are any unmapped dune fields on Mars. To find potential candidates we can apply our model to the full unlabeled dataset (~76,000) 256 x 256 image tiles. We had to divide these into 38 files containing 2,000 tiles per file due to RAM and disk limitations. A shortcut to the image files can be found here. The model 'deepdunes_model.hf' can also be downloaded (or a shortcut can be made) here

import keras
import numpy as np
#Load model
model = keras.models.load_model('/content/gdrive/My Drive/deepdunes_model.h5')

#function for loading in each 2000 image .npz (38 total files)

n = np.arange(37)
file_path_list = []
for i in range(len(n)):
  filename = '/content/gdrive/My Drive/mars_dunes_full_ds/mars_dunes_full_ds_'+str(i)+'.npz'
  file_path_list.append(filename)

We now use our model to 'detect' pixels that may correspond to unmapped dune fields. This is only a preliminary result and needs more work!

import numpy as np

predicted_labels_list = []
predicted_data_list = []

for i in np.arange(len(file_path_list)):
  print(file_path_list[i])
  loaded = np.load(file_path_list[i], 'r')
  testing_data = loaded['data']

  #normalize data
  mean = testing_data.mean(axis=(1,2))[:,np.newaxis,np.newaxis]
  std = testing_data.std(axis=(1,2))[:,np.newaxis,np.newaxis]
  testing_data_normal = (testing_data-mean)/std

  #add dimension of 1
  testing_data = np.expand_dims(testing_data,axis=3)
  testing_data.shape

  prediction = model.predict(testing_data)
  
  # thresholding probability
  labels_pred = prediction  
  labels_pred[prediction>=0.5]=1
  labels_pred[prediction<0.5]=0

  plot_mars_dune(testing_data[:,:,:,0],labels_pred[:,:,:,0],'Possible Dune Fields')

  #clear memory/disk so the next file can be read in. See how to do this?
  del loaded
  del testing_data
  keras.backend.clear_session()

/content/gdrive/My Drive/mars_dunes_full_ds/mars_dunes_full_ds_0.npz

/content/gdrive/My Drive/mars_dunes_full_ds/mars_dunes_full_ds_1.npz

/content/gdrive/My Drive/mars_dunes_full_ds/mars_dunes_full_ds_2.npz

/content/gdrive/My Drive/mars_dunes_full_ds/mars_dunes_full_ds_3.npz

/content/gdrive/My Drive/mars_dunes_full_ds/mars_dunes_full_ds_4.npz

Future work

Our training dataset included 454 hand-mapped dune fields. Using our model, we identify additional regions which may correspond to dune fields over the same region. Examples of dune fields that our model identified that were not hand-mapped are plotted above.

Next steps will involve transforming the predicted/possible dune fields back into geographic coordinates for visual inspection in GIS software. This will involve stitching image tiles together or creating polygons from the mapped dune field candidates and QC.

Summary

We have developed a deep learning workflow that utilized the UNet convolutional neural network architecture to first test the feasibility of automated detection of dune fields on Mars from satellite images. Our final model predicts the location, geometry, and extent of dune fields with a validation accuracy of 0.97. We then apply our model to the full dataset and identify areas that may correspond to additional unmapped dune fields on Mars. This project provides a proof-of-concept that CNNs may provide an automated way of locating and mapping the extent of dune fields on the Martian surface.